1. Field of the Invention
The invention relates to a computer-aided method of contact-free video-based determination of the direction of view of an eye of a viewer for the eye-controlled man-computer interaction in accordance with the cornea reflex method of detecting view-direction dependent eye vectors between the detectable center of the pupil and reflex points which are generated on the cornea by aimed infra red irradiation and which can also be detected, with calibration with respect to at least one calibration point which can be depicted on a video screen for defining a view direction function and a recalibration to be performed in dependence of the generation of image objects and a plurality of reflex points for correcting systematic errors of measurement, and to an apparatus for practicing the method.
2. The Prior Art
The continuing integration of information technological systems offers computer users a wide spectrum of novel services and access to almost unlimited multi-media data sources. The increased possibilities of utilizing computers lead to increasingly higher demands in respect of the user interface. Solutions are being sought which would enable users of vastly differing initial knowledge to completely control computers. Among these, particular importance is to be attached to a progressive user interface which places more emphasis on human perceptiveness and a natural repertoire of possible interactions than is the case with current graphic interfaces including a mouse and a keyboard. The spatial presentation of information on a 3D-display not requiring spectacles, support of active vision by influencing the presentation of information by way of the head position and the view direction and non-command based interactions anticipated by an interface agent on the basis of the viewing behavior are key elements.
Different proposals based upon novel presentation and interaction techniques have been made as regards a viewer's desire for quick cognitive orientation, complete hold on and personal control over the novel systems. Among these are, in particular, intuitive direct manipulation by active viewing and manual gestures. As in natural viewing, the viewing perspective may change as the head moves. In this manner, for inspecting hidden planes, objects in the foreground may be laterally pushed away. Additional visual interactions become possible by detection of the viewing direction or of a fixation point. For instance, looking at a media object may be utilized for the fading-in of context information. A synthetic depth of field coupled to the fixation point may be useful to the user to concentrate his attention on the selected media object. Morever, the viewing direction may be utilized to address, move or zoom video screen objects by visual fixation (eye mouse function).
Methods of measuring the viewing direction have in the past been primarily developed for laboratory experiments or for purposes of medical diagnostic, rather than for application in interactions between man and computer. The technological possibilities have in essence been known since the seventies and may be distinguished between contact-free and non-contact-free methods. The non-contact-free methods, for instance, require the attachment of skin electrodes to the face of a viewer or carrying special spectacles or contact lenses. For man-computer interaction, video-based techniques are suited best. To this end, an image of the eye is recorded by a video camera, preferably filling the format, and is evaluated in accordance with different criteria. For purposes of a contact-free measurement the view measuring camera is set up close to the video screen. A tele optic system, optionally provided with a tracking system which aims the camera at the eye even at movements of the head, ensures a format-filling image of the eye. Movements of the pupil are evaluated relative to a reference system such as, for instance, the center of the camera image. For precise measurements, the eye is illuminated by weak non-blinding infrared light to illuminate the eye region and selectively to generate reflections at border layers, such as, for instance, at the surface of the eye and at the rear surface of the cornea (Purkinje reflexes). The position of the reflection relative to the center of the pupil are evaluated. The strongest reflection occurs at the surface of the cornea. The so-called cornea reflex method (see “Bildverarbeitung durch das visuelle System”, A. Korn, Springer Verlag, 1982, pp. 126 ff.) drawing on the fact that the cornea reflects impinging light which can be detected as a bright spot on the surface (cornea reflex), enables precise contact-free measurements over an extended period of time without recalibrations. The view measuring camera is aimed at an eye of the user by signals from a head tracker and follows the eye even at movements of the head.
During the course of the image evaluation, first the pupil is localized by iteratively approaching the pupil with a line of annular contour in accordance with the least-square method. Thereafter, the reflex point is determined as the center of the largest very light area in a somewhat broadened environment of the pupil, by a simple threshold value operation. In this manner, the so-called eye-vector is being defined which points from the center of the pupil to the center of the reflex point. At ever change of the view direction the eye-vector between the centers of the pupil and of the reflex point changes continuously relative to the view direction. The value is a function of the geometry of the individual eye (radius of curvature of the cornea and of the eyeball) and may be defined by a continuous function (view direction function). The view direction function has to be determined by the calibration. Following the calibration, at which the user must sequentially fix, e.g., five calibration points as pointed objects on the video screen, the eye-vectors may be calculated relative to the calibration points. The more calibration points used in a predetermined viewing sequence and the longer the fixation period of any given user lasts, the more precise will be the determination of the eye-vectors. In the known method, this complex operation must be performed at the beginning of every individual application and this necessitates a high degree of attention and great patience on the part of the user. During fixation of a given calibration point an interpolation model establishes a connection between the X and Y video screen coordinates of the calibration points and the corresponding x and y values of the measured eye-vectors (in the camera image). The individually adjusted parametric values of the model are stored and during view measurement serve exclusively for interpolating the values of the video screen coordinates during fixation of any points of the video screen. Depending upon the number of calibration points and the user's ability to concentrate, the calibration process takes at least one to two minutes.
Movements of the head lead to changes in the eye-vector and, hence, of the calculated view direction, even if the user retains the fixation point. The reason for this change is that the eyeball has to be rotated to ensure alignment of the line of sight with the (stationary) fixation point. Since the view camera has to be inclined and pivoted to retain the eye in the center of the image when the head is moved, the surface normal of the camera sensor changes also. This (systematic) measuring error can be effectively corrected by a newly developed transformation technique (see DE 197 36 995.2 A1 or the paper “Determination of the Point of Fixation in a Head-Fixed Coordinate System”, Liu, 14th Intern. Conf. On Pattern Recognition, 16-20 Aug., 1998, Brisbane, Australia). It utilizes a movable coordinate system related to the position of the user's head. For transformation of the measurement data, the center of the eye is selected as the coordinate source. The view direction calculated in the camera image can be converted to the 3D-coordinates of the fixation point by rotation, scaling and translation. The degrees of freedom of the measuring system and of the display are not limited. In this manner, only a few transformation steps are required to compensate for movements of the head. The mentioned explanations have been taken from the paper “Augenmaus & Co.—Entwicklung einer neuartigen Anwenderschnittstelle für Multimedia-Computer” Liu, Pastoor, ZMMS-Spektrum (ISBN 3-932490-52-5) 1999; Vol. 8, pp. 56-70, and from the final report on the support project 01BK410/8 (BLICK), 30.04.99, pp. 57-61.
Before discussing individual calibration methods in greater detail hereafter, a definition of the term “calibration” will first be interjected. It is because confusion often arises in connection with the use of this term as it incorrectly also used for compensation and correction processes in measuring methods. The calibration process is to be understood as a “calibration process” of a measuring arrangement for determining a relative reproducible connection between output values and individual input values. By contrast, the calibration process has to be defined as “gaging process” for examining and defining absolutely the measuring precision and certainty of measuring instruments. Compensation processes, such as, for instance, compensating for the head movement when defining the view direction or correction processes, for instance correcting the measurement value at a change in distance of the user within the spatial depth relative to the view measuring camera, are not calibration processes. To speak of these as “post or recalibrating” is not correct in terms of measuring technology.
In some prior art applications in the field of a video-based determination of the view direction no explicit calibration processes is provided. In a camera with a view-controlled autofocus according to U.S. Pat. No. 5,678,066 the individual user cannot perform any calibration process. In accordance therewith, the range of selection can be segregated into three regions only, corresponding to a very coarse determination of the view direction. A camera of the same type provided with an improved lens-mirror arrangement and a more advanced detection method is described in U.S. Pat. No. 5,875,121. In this case, too, no individual calibration is possible; view measuring distinguishes only in four main directions “left”, “right”, “above” and “below”. An improvement of the measuring precision with calibration results from the head of the user being fixed, so that only the physiological properties of the eye systematically affect any error. Such a measure has been in the context of a method of measuring the visual field of patients in U.S. Pat. No. 5,220,361. For an at least partial compensation of the systematic measurement error a total of four reflex points are generated on the cornea; in this fashion the distance between the eye and the sensor can be determined. However, the individual geometry of the eye (e.g. the radius of the eye) is not taken into consideration. Overall it can be said that in methods which do not include calibration the view direction can be measured with a substantially reduced precision and that the results yielded can only be coarse estimates of the view direction. This is, however, insufficient for many applications. These are “simple systems” without individual calibration, which rely on the generalized parameters of “standard eyes”. Sufficient precision can only be attained if the eyes of the user are identical to the standard eyes.
This distinguishes them from the “elaborate systems” in which a time-consuming and work-intensive calibration is performed for each individual user. They differ in the number and arrangement of the applied calibration points. In general, the precision increases with the number of applied calibration points. U.S. Pat. No. 5,634,141 discloses a camera with a view-controlled autofocus system and in which calibration may be performed individually. To this end, two horizontally aligned calibration points are provided at the viewer of the camera. To generate reflex points around the eye of the user which serve to measure his individual pupil diameter and to calculate the axis of the eye as a result of different assumptions on the basis of standardized eyeball parameters (causality examination), two of six infrared light diodes disposed around the viewer are used in pairs, depending upon the position of the camera and the condition of the user. Here, too, the precision of the calibration is only adequate to define visual ranges, in this instance three ranges in a horizontal direction. In order somewhat to reduce the complexity of the calibration, the camera provides storage of different sets of calibration such as, for instance, for different users or for users wearing or not wearing spectacles. In case of need, the corresponding set of data may then be retrieved. Aside from the fact that for purposes of exact measurements it is extremely important to retrieve the set of data appropriate for a given user, automatic adjustment to individual eye data is not guaranteed. In order to take changes into account it is necessary to perform a new calibration.
The state of the art upon which the invention is based is described in German laid-open patent specification DE-OS-197 31 303 A1. It discloses a method for contact-free measuring, correctively taking into account large movements of the head or eyes, the view direction. In the method, the coordinates of spider lines for the center of the pupil and at least one corneal reflection of a single user are calibrated in respect of at least one calibration point (thirteen calibration points for the utmost precision). To this end, the appropriate number of calibration points are displayed on a video screen. The user must then, for a sufficient length of time and in a predetermined sequence, fix these calibration points. This is subject to automatically controlling whether he has looked at the fixation target for a sufficient length of time and whether he is following it. If the user does not perform properly, he will be prompted to repeat. On the basis of different mathematical models and approximation processes a transformation matrix (view direction function) will then be derived from the attained measurement data of the pupil position, corneal reflection and the coordinates of the predetermined calibration points. Furthermore, the known method provides for an “automatic adjustment of the calibration data” at horizontal and vertical changes in the distance between the eyes of the viewer to the video screen and to the image measuring camera. This is, in fact, a compensation or correction of significant movements of the head and eyes in the plane parallel to the video screen. An “automatic post-calibration” is disclosed as well. This is a purely mathematical adjustment at changes of the distance between the eye to the camera lens (in the depth dimension) by way of the correspondingly changes autofocus setting. However, in the sense of the definition set forth supra neither correction process can be correctly classified under the term “calibration”. It is different, however, in respect of the dynamic “re-calibration” which is also provided. To correct systematic errors of measurements, for instance of changes in the diameter of a pupil, a comparison of desired and actual values relative to previously defined objects is repeatedly performed on the video screen for each individual user. In case a permissible measurement error is exceeded recalibration occurs automatically. Aside from the fact that it is not clear how the system recognizes the excess, this dynamic recalibration constitutes a significant annoyance to the user who will always and by way of surprise be subjected—even in active use—to a new calibration process.
Hence, it may be said overall that known calibration processes in methods of defining a view direction significantly curtail the comfort of a user. On the one hand, he is required to exercise a high degree of concentration and great patience during fixation of the calibration points, particularly moving ones. On the other hand, different users of a view direction measuring system must, before use, each go through an individual complex calibration process in order to ensure his individual calibration data.